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ABSTRACT 

Continuing demand for the use of Unmanned Aircraft Systems (UAS) has put 
increasing pressure on operations in civil airspace. The need to fly UAS in the 
National Airspace System (NAS) in order to perform missions vital to national 
security and defense, emergency management, and science is increasing at a rapid 
pace. In order to ensure safe operations in the NAS, operators of unmanned aircraft, 
like those of manned aircraft, may be required to maintain separation assurance and 
avoid loss of separation with other aircraft while performing their mission tasks. 
This experiment investigated the effects of varying levels of automation on UAS 
operator performance and workload while responding to conflict resolution 
instructions provided by the Tactical Collision Avoidance System II (TCAS II) 
during a UAS mission in high-density airspace. The purpose of this study was not to 
investigate the safety of using TCAS II on UAS, but rather to examine the effect of 
automation on the ability of operators to respond to traffic collision alerts. Six 
licensed pilots were recruited to act as UAS operators for this study. Operators were 
instructed to follow a specified mission flight path, while maintaining radio contact 
with Air Traffic Control and responding to TCAS II resolution advisories. Operators 
flew four, 45 minute, experimental missions with four different levels of 
automation: Manual, Knobs, Management by Exception, and Fully Automated. All 



missions included TCAS II Resolution Advisories (RAs) that required operator 
attention and rerouting. Operator compliance and reaction time to RAs was 
measured, and post-run NASA-TLX ratings were collected to measure workload. 
Results showed significantly higher compliance rates, faster responses to TCAS II 
alerts, as well as less preemptive operator actions when higher levels of automation 
are implemented. Physical and Temporal ratings of workload were significantly 
higher in the Manual condition than in the Management by Exception and Fully 
Automated conditions. 
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I INTRODUCTION 

Continuous demand for the use of Unmanned Aircraft Systems (UAS) has put 
increasing pressure on airspace operations in civil airspace. This demand is driven 
by two main advantages that UAS have over manned aircraft, perceived cost 
efficiency and the minimization of risk to pilots’ lives (Gertler, 2012). The need to 
fly UAS in the National Airspace System (NAS) in order to perform missions vital 
to national security and defense, emergency management, and science is increasing 
at a rapid pace due to the foreseen advantages of their use. In addition to limiting 
UAS usage for civilian applications, current Federal Aviation Administration (FAA) 
restrictions on UAS access to the NAS constrain the U.S. military’s ability to fulfill 
regular training requirements to prepare UAS operators for combat (DoD UAS 
ExCom NAS Access Working Group, 2010). 

In order to ensure safe operations in the NAS, operators of unmanned aircraft, 
like those of manned aircraft, may be required to maintain separation assurance and 
avoid loss of separation and conflicts with other aircraft while performing their 
mission tasks. A commonly used conflict avoidance measure for manned aircraft is 
TCAS II, or the Tactical Collision Avoidance System. TCAS II is a transponder 
based system that provides Traffic Advisories (TAs) to alert the pilot of incoming 
traffic, and Resolution Advisories (RAs) to provide pilots with instructions for 
avoiding conflicts within a five second time limit (FAA, 2001). Resolution 
Advisories provided by TCAS II are limited to vertical maneuvers only; examples 
include “Climb, Climb” and “Descend, Descend”. While the FAA has ruled TCAS 

II alone to be unacceptable for UAS flight due to latencies that are inherent to UAS 
operation and the differing flight characteristics from manned aircraft (FAA, 2011), 
it is reasonable to expect that a TCAS II - like system could be part of a layered 
solution involving an integrated traffic display and conflict alerting system. 

Those latencies present in UAS operations may create the need for automation 
to assist operators in responding successfully to conflict alerts, regardless of the 
conflict avoidance system, or suite of systems, used to provide them. The wide 
variability of automation capabilities in current unmanned systems raises the 
question about the effect of human-automation interaction on the ability of 
operators to respond safely and timely to conflict alerts. Some present day UAS 



(e.g. MQ-1 Predator) require manual control of the flight control systems, and 
operator tasks closely resemble those traditionally associated with manual flying. 
Other UAS (e.g. RQ-4 Global Hawk) are highly automated, with operators flying in 
pre-programmed waypoint-to-waypoint navigation mode under a supervisory 
capacity. Most current systems, however, fall somewhere in between fully manual 
and fully automated with a wide variety of partially-automated control and 
navigation interfaces. When latencies are present in the control loop, it becomes 
important to know whether operators can respond quickly enough to potential 
conflicts given the type of control input and automation capability present. 

This experiment investigated the effects of varying levels of automation on UAS 
operator performance and workload while responding to conflict resolution 
instructions provided by a Tactical Collision Avoidance System II (TCAS II) during 
a UAS mission in high-density airspace. The purpose of this study was not to 
investigate the safety of using TCAS II on UAS, but rather to examine the effect of 
automation on the ability of operators to respond to traffic collision alerts provided 
by a commonly used system. 

2 METHOD 

2.1 Participants 

Six pilots were recruited to participate in this study. All were males (averaged 
29.5 years) with an average of 2727 flight hours. Total flight hours ranged from 250 
to 5300 hours. No pilots reported military or UAS flight experience. Eligibility was 
limited to participants who had normal or corrected to normal vision and were under 
40 years old. Participants were required for approximately seven hours each and 
were compensated for their participation in the study. 

2.2 Displays Setup 

Participants were given two computer monitors to observe and manipulate: the 
Multiple-UAS Simulator (MUSIM) on the right, and the Ames 3D Cockpit 
Situation Display (CSD) on the left (Figure 1). 

Multiple-UAS Smulator (MUSM). This experiment used the Multiple-UAS 
Simulator (MUSIM), a full description can be found in Fern & Shively (2009). The 
current simulation configuration of MUSIM differed only slightly in that it utilized 
a 1:1 operator to vehicle interface with a generic fixed wing flight control model 
input with generic Mid-Altitude Long Endurance (MALE) UAS parameters. 
Ownship airspeed was fixed at 90 kts for the entire experiment. MUSIM was 
separated into four Graphical User Interfaces (GUIs): a map display indicated the 
position and flight path of the UAS in purple waypoints, a multi-function display 
(MFD) indicated UAS status and behavior, a TCAS II (Tactical Collision 
Avoidance System II) alert box provided textual and auditory Traffic Advisories 
(TAs) and Resolution Advisories (RAs), and a timer. 




Figure 1 Displays set up with Multiple-UAS Simulator (MUSIM) (right) and Ames 3D Cockpit 
Situation Display with traffic information and TCAS II alerts (left). MUSIM interfaces include the 
map display (left), MFD (top right), TCAS alert box (lower right) and timer (far right). 


Ames 3D Cockpit Stuation Display (CSD). The Ames 3D Cockpit Situation 
Display was used to display TCAS II information in its basic 2D planar view (for a 
full description see Granada, Dao, Wong, Johnson & Battiste, 2005). The CSD had 
an ownship-centric view of surrounding airspace and utilized TCAS II symbology 
to alert operators of potential collisions. Participants were able to adjust the 
horizontal viewing distance from 10-640 nm, though no other manipulations were 
allowed on the CSD during this experiment. 

2.3 Experimental Design and Mission Details 

A within-subjects design was used to study operator perfonnance and workload 
measures while flying a signal intelligence mission using a MALE UAS in high 
density Southern California TRACON airspace (LAX terminal area). Four different 
levels of automation for responding to TCAS II RAs were counterbalanced across 
four different mission flight paths. 

Levels of Automation. Operators were given four different levels of automation 
to assist them in responding to TCAS II RAs: Manual, Knobs, Management by 
Exception and Fully Automated. In the Manual condition, operators flew the UAS 
in a waypoint-to-waypoint control mode. This was the baseline control mode for all 
conditions. Flight paths were edited by clicking each waypoint individually to 
activate the editing function. Once in editing mode, new altitudes were input 
manually and waypoints were clicked and dragged to new locations. In order to 
respond to an RA in the Manual condition, the operator would click on the next 
waypoint, manually input the new altitude, and commit the change. In Knobs, 
operators utilized the MFD Control and Status Page to quickly edit the altitude of 
the aircraft when an RA was received. Use of the Knobs input automatically applied 
the altitude change to the next waypoint on the aircraft’s path, and was introduced 
as a “quick response” control option similar to dialing an analog knob in a cockpit. 
To change additional waypoints on the flight path, operators had to use the Manual 
control mode. In Management by Exception, participants either accepted or rejected 



an automated altitude edit in the MUSIM TCAS II alert box when RAs were 
received. In the Fully Automated condition, altitude changes in response to RAs 
were automatically applied, with feedback provided to the operator of the altitude 
change in the MUSIM TCAS II alert box. All automated responses were in 
compliance with the RA instructions. In both the Management by Exception and 
Fully Automated conditions, all other route edits not in response to RAs were done 
through the Manual control mode. 

Missions. Four training and four experimental missions were developed for this 
experiment. Training scenarios were 10 minutes long, and provided rerouting 
practice for the operators in the four different levels of automation. Experimental 
missions were 45 minutes long and differed only in the assigned waypoints and 
altitudes in the flight path that the operator was instructed to follow. All missions 
included TCAS II alerts that required operator attention and rerouting for RAs, 
though timing of conflicts and severity differed between missions to reduce 
predictability. 

Mission Objectives Participants were instructed to fly a signal intelligence 
mission with a MALE UAS in Southern California TRACON airspace with three 
mission objectives: 1) to fly the assigned mission flight path as closely as possible, 
2) to respond to TCAS II alerts for collision avoidance by either climbing or 
descending in accordance with the Resolution Advisory, and 3) to communicate 
with Air Traffic Control (ATC). The first objective required operators to fly through 
predetermined mission flight paths while maintaining set altitudes at each waypoint 
and remaining as close to the original flight path as possible. The second objective 
required operators to monitor the TCAS II alerts on both the MUSIM and CSD 
displays and reroute the UAS’s path to avoid conflicts while staying as close as 
possible to the original route. The third objective required operators to maintain 
radio contact with ATC while flying their route; tasks included calling in altitude 
requests, flight path edits, waypoint check-ins and responding to any received ATC 
communications. 

2.4 Procedure 

Participants were required to fill out an informed consent form and a 
demographic survey. Training was given to the operators before each experimental 
condition, with workload and situation awareness probes administered throughout. 

Training Sessions. Operators were given a short briefing introducing MUSIM, 
the CSD and mission objectives after completing paperwork. Self-paced PowerPoint 
slides were provided before each experimental session detailing how to edit 
waypoints and altitudes appropriately for each condition. Training scenarios were 
then completed and lasted 10 minutes. 

Experimental S&ssions. The experimental sessions were blocked by level of 
automation and flight path using a Latin Square. Participants completed four 
experimental missions during the simulation. In each mission, operators received 
workload probes after each RA, administered verbally by the experimenter. Before 
each scenario, participants were given a practice mission to familiarize themselves 



with the level of automation present. After each scenario, participants completed a 
NASA-TLX (Hart & Staveland, 1988) to measure workload and a 10-D SART 
(Situation Awareness Rating Technique; Selcon, Taylor, & Koritas, 1991) to 
measure situation awareness. At the end of the day, operators completed a Post- 
Simulation Questionnaire asking more in depth questions on workload and SA. 

3 MEASURES 

3.1 Objective Performance 

Response Time. Response Time (RT) to RAs was measured in seconds from 
when the TCAS II alert was given until the operator committed a flight path change 
in the Manual and Knobs conditions. In the Management by Exception condition, 
RT was measured in seconds from when the operator was alerted of the RA to when 
they responded by clicking either “Accept,” or “Reject,” on the MU SIM TCAS alert 
box. If the operators did not respond within 5 seconds, the alert box timed out and 
automatically adjusted the altitude of the UAS. Reaction Time was not measured in 
the Fully Automated condition as operators did not need to respond to the RAs. 

Operator Response Rate. Operator Response Rate measured the percentage of 
RAs that operators responded to (correctly or incorrectly) out of the total that 
occurred. 

Compliance Rate. Compliance Rate was measured as the percentage of RAs that 
operators correctly complied with out of the total number that occurred. 

Pre-Emptive Response Rate. Pre-Emptive Response Rate was measured as the 
percentage of times that the operator began a route edit in anticipation of an RA out 
of the total number of RAs that he responded to. 

3.2 Subjective Ratings 

A/AS4 TLX. Workload was measured post-scenario with a 10-point NASA TLX. 
Participants rated their workload on six dimensions: Mental Demands, Physical 
Demands, Temporal Demands, Performance, Effort and Frustration. 

Additional Measures Additional Subjective Ratings were collected, though not 
discussed in this paper. Additional measures include the 10-D SART, workload 
probes administered during trials, a post-trial questionnaire and a post-simulation 
questionnaire that were given to further measure workload and situation awareness. 

4 RESULTS 

The data were analyzed using a one-way repeated measures analysis of variance 
(ANOVA) with Levels of Automation as the independent variable. Post hoc 
analyses utilized Bonferroni pair-wise comparisons. The results are organized by 
type of measure. 



4.1 Objective Performance 

Response Time. Response times to TCAS II alerts were found to be significantly 
faster in the Management by Exception condition ( M= 3.45; SE= .35) compared to 
both Knobs (M= 12.20; SE= 2.56) and Manual (M = 10.75; SE= 1.44), F( 2, 12) = 
8.408, p< .05 (Figure 2). 
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Figure 2 Response Time to TCAS II alerts by Level of Automation 


Operator Response Rate. There was not a significant difference in operator 
response rates across the different conditions, F( 2, 12) = 2.602, p= .115. However, 
there appeared to be a trend toward higher response rates in Management by 
Exception (M = 95.92; SE= 2.64) than in both Manual (M = 85.86; SE= 4.09) and 
Knobs ( M= 74.40; SE= 10.25) (Figure 3). 
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Figure 3 Response and compliance rates by level of automation. 


Compliance Rate. Compliance rates for Management by Exception (M = 89.71; 
SE= 6.04) were significantly higher than both Manual ( M= 74.40; SE= 10.58) and 
Knobs ( M = 63.78; £E= 12.27), F( 2, 12) = 7.233, p < .01 (Figure 3). Several 
reasons were noted for noncompliance; operators did not feel a collision would 
occur, horizontal route edits were performed instead of vertical, and accidental 
inputs of wrong altitudes were made. 

Pre-Emptive Response Rate. Operators made significantly more pre-emptive 
responses in the Manual condition (M = 6.21; SE= 2.96) and the Knobs condition 
(M = 42.09; SE= 8.20) than in Management by Exception {M = 6.20; SE= 2.96), 
F(2, 12) = 8.705, p<. 01. 

4.2 Subjective Ratings 

A//4S4 TLX. Significant differences between the levels of automation were found 
in the physical and temporal dimensions of workload, F( 3, 18) = 3.358, p< .05, and 
F(3, 18) = 4.078, p < .05, respectively (Figure 4). Ratings of physical workload 
were significantly higher for Manual (M = 3.43; SE= 1.02) compared to Knobs (M 
= 2.57; SE = 1.04), Management by Exception ( M = 2.21; SE = .69), and Fully 
Automated (M = 2.29; SE = .68). Ratings on temporal workload were significantly 
higher for Manual {M = 6.00; SE= .59), compared to Knobs (M = 3.50; SE= .29) 
and Fully Automated ( M = 3.93; SE = .93). Ratings of temporal workload in 
Management by Exception ( M= 4.43; SE= 1.09) were not found to be significantly 
different than any of the other levels of automation. 
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Figure 4 NASA-TLX ratings on the Physical and Temporal dimensions of workload by level of 
automation. 
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DISCUSSION 


In order for UAS to safely operate in the NAS, they must be able to successfully 
avoid conflicts and respond appropriately when collisions are imminent. This study 
examined the effects of varying levels of automation on the ability of UAS 
operators to respond to traffic collision alerts in high-density airspace. The results 
indicate that response times and compliance rates for unmanned aircraft operating 
with lower levels of automation could be unacceptable in the NAS environment. 
The required five-second or less response time was exceeded in both the Manual 
and Knobs levels of automation by more than five seconds on average. In the event 
that a UAS is relying on satellite communication and data links, it will have 
additional operational delays of up to two seconds. This could create an aircraft 
response time to resolution advisories of 12 seconds or more; over twice the current 
five-second standard for manned aircraft. 

Current regulations allow for noncompliance with TCAS II alerts only when 
pilots have the conflicting aircraft in visual sight and can ensure separation (FAA, 
2001). Although difficult to find accurate reports for actual TCAS II compliance 
rates for manned aircraft, compliance with RAs has been estimated between 50 - 
60%. This is supported by Olson & Olszta (2010), who found a compliance range of 
41 - 59%, and Pritchett & Hansman (1997) who reported approximately 61% of 
pilots complying within the five second limit, and a compliance range of 33 - 71% 
when including pilots who began to edit for RAs preemptively or late. 

Compliance rates across levels of automation in this experiment often exceeded 
these previously reported rates, with averages for Manual, Knobs and Management 
by Exception being 74%, 64% and 90%, respectively. Possible reasons for this 
difference include among others: a lack of an out-the-window view to be used in 
visual separation (in which application of the FAA regulation to UAS operation 
should result in 100% compliance); unfamiliarity with the flight characteristics of a 
MALE UAS causing a reliance on automated, opposed to personal, collision 
avoidance judgments; taking longer than five seconds to decide on a course of 
action in the Management by Exception condition (forcing the automation to 
comply with the resolution advisory); trust in automation; and ease of use for 
conflict avoidance in the higher automated conditions. Further research is required 
to better understand the benefits of automation for increased safety and collision 
avoidance while operating in the NAS. 

Interestingly, the Manual condition was found to perform better than the Knobs 
condition overall with a lower average response time, and a higher average response 
and compliance rate, despite Knobs being introduced as a “quick response” control 
input. The lower performance for Knobs goes with a trend of more preemptive, or 
anticipatory, responses made in that condition. Operators remarked on how the 
Manual and Knobs conditions took “too long,” to perform the edits required for 
collision avoidance, and would begin to preemptively edit before an RA was given 
in order to avoid the impending collision. 

Overall, slow response times for Manual and Knobs, combined with 1.5 times 
greater compliance rates for Management by Exception illustrate the need for some 



level of automation to assist UAS operators in responding quickly and appropriately 
to collision avoidance alerting while operating in the NAS. However, the lower 
workload scores for Management by Exception compared to the Fully Automated 
condition indicate a need for the operator to remain on the loop and capable of 
overriding the automation when necessary. Continued research on the effects of 
human-automation interaction on the safe operation of UAS in the NAS is needed if 
the demands for civil and public UAS operations are to be met. 
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